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ABSTRACT 

This paper reports on an exploratory pilot study at Network 
South Enterprises (NSE) (Manitoba), a non-profit, community-based 
organization with the purpose of providing employment services for adults 
with mental disabilities. The study used a focus group, an open-ended 
questionnaire, and a 36-item InfoMMS scale to determine the motivational 
appeal of NSE print materials and contents. Seven employers took part in the 
pre-benchmarking phase of the study to determine any difficulties with the 
current performance levels of NSE corporate information materials . Another 
seven employers were later used to determine the performance levels and the 
effects of newly developed treatment materials and contents. The study 
provided vital internal pre-benchmarking stage pretest data for: (1) 

estimating the reliability of the InfoMMS administration; (2) estimating the 
performance levels or usefulness of existing corporate materials; (3) 
clarifying design, development, and use of future corporate information 
materials and preferences; and (4) providing a ready bank of baseline control 
data against which future materials and associated benchmarking partner data 
can be compared. (MES) 
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INTRODUCTION 

When you choose to read a printed page, you think there is some intrinsic quality 
about it, and this conception of the "quality" is part of a motivational structure which 
establishes priorities about your reading experience. To ask what an individual wants to read 
is to a large extent to ask what this person wants to experience. When this journey must be 
planned, the effective designer asks what catches a reader's eye or what does a reader find 
useful, memorable and gratifying, for these are the essential features which nourish quality 
and give rise to a climate for reading. 

Many businesses, corporate conglomerates, profit and non-profit organizations are 
just beginning to reflect a renewal of interest to examine or incorporate these motivational 
aspects in the process of designing their promotional material, and or parallel practice. They 
have a huge role to play in marketing themselves and their services to their stakeholders. 
They can be quite boisterous and quick about letting us know about what marketing media 
they use to dispense such information. Brochures, employer information packages, posters, 
newspaper ads, video commercials are not unusual, and more recently web sites among them 
are frequently used. They are a little quieter, however, when it comes to telling us about 
whether the contents of their messages actually have any appeal to their stakeholders. Not 
Network South Enterprises, Inc.(NSE). That's because NSE has been busily engaged in 
upgrading their marketing initiatives in the hopes of improving their services. They not only 
want employers to receive their messages, but also expect employers to perceive these 
messages as motivationally appealing. As such, this NSE Business Advisory Council 
volunteer research consultant was specifically charged to focus on improving what worked, 
and provide recommendations to rectify what didn't while ensuring that the needs of NSE are 
best being served. 

NSE is a non-profit, community based organization begun in 1991. NSE is funded 
primarily through the generous support of Human Resources Development Canada and 
Manitoba Department of Family Services. It was founded with the principal purpose of 
providing employment services for adults with mental disabilities. Their mission, then, is to 
support adults with mental disabilities to work and participate in the community, where they 
are respected and rewarded for their efforts, skills and accomplishments. They do this by 
working cooperatively with employers throughout the city of Winnipeg who share the same 
vision. Together they are making significant inroads in achieving the mission. Over 600 
disabled workers were placed to date. That's a tremendous achievement and commitment to 
action. 

This exploratory pilot study was limited to using a focus group, an open-ended 
questionnaire and a 36-item InfoMMS scale. The latter scale attempts to document the 
motivational appeal of print reading materials and contents in an objective format so that the 
researcher is able to understand and interpret the employer's experience as accurately as 
possible. As such, seven employers took part in the prebenchmarking phase of this study to 
determine any difficulties with the current performance levels of NSE corporate information 
materials. Another seven employers were later used to determine the performance levels and 
the effects o f n ewly d eveloped t reatment m aterials a nd c ontents. P urposive o r j udgmental 
sampling was employed because NSE Business Advisory Council members appealed to their 
own judgement and experience to select people for the sample who would best attend to the 
research questions of the study. Owing to these limitations, the results of this inquiry should 
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be used primarily to inform and stimulate future research. As indicated above, the study used 
a two foci approach to internal benchmarking and product improvement. Quantitative and 
qualitative. The former using objective instruments, and the latter employing an open-ended 
questionnaire, a focus (reference) group of employers and a Business Advisory Council from 
NSE to guide the design, development and evaluation decision-making process. The report is 
subdivided into the pre-benchmarking stage and benchmarking stage. An analysis and 
discussion of results, observations and concluding comments are provided. 

PRE-BENCHMARKING STAGE 

The Research Design 

The prebenchmarking stage o f research w as c onducted i n M arch 23,1 999 u sing a 
small focus group of employers and a posttest only group design. This was required to set the 
stage for development. A Focus Group session was held to establish baseline measurement 
data of existing NSE corporate information materials and contents from an employer's 
perspective. The measurement data in this stage is called baseline because they show the 
employers' natural responses to existing NSE materials and contents without experimental 
manipulation. The Focus Group also responded to three open ended survey questions about 
the existing materials. What did they like best about the existing NSE Employer Information 
Package? What did they like least about it? What recommendations did they have for 
improvement? A face to face discussion followed immediately afterward, the results of 
which were tabulated in Table 4 of the Appendix. This researcher examined over 30 
Corporate Information Brochures in the interim. 

The benchmarking stage of research was conducted in November 29, 2000 using a 
small alternate group, a small experimental group and a posttest two-group crossover or 
counterbalanced design. In the counterbalanced design, two treatments (i.e., 2 newly 
designed brochures) were administered in November 29, 2000 to each employer. The order 
of treatments was varied across subjects to control possible confounding of treatment effects 
with order effects. That is, employers received treatments in either of the sequences X\, X 2 or 
X 2 , X] depending on what side of the table they sat on when they walked in to the 
boardroom. Each treatment was followed up with a means measurement of the study's 
dependent variable, the InfoMMS scale. Comparisons were then made on the effects of these 
two treatments using the March 23, 1999 baseline group data as the "control." The basic 
pattern of the counterbalanced design can be diagrammed as follows: 

Period 1 

Period 2 

Where: 



The two periods in this pilot test were the same lengths (of time). Thus, both treatments 
were administered in each time period rendering the comparisons between treatments free of 



X 



1 Alternate Brochure 



x 



2 Experimental Brochure 



o 

o 



X-2 Experimental Brochure ^ O 

Xl Alternate Brochure ^ O 



X] = Alternate treatment brochure 

X 2 = Experimental treatment brochure 

O = means measurement or observation (posttest) 
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period effects. In other words, rotating the sequence in which different treatments were 
presented to subjects was done in an effort to control for extraneous carry-over variables, 
such as practice and fatigue effects. Relevant to the choice of design under discussion, Borg 
and Gall (1989) indicate the following advantages of employing counterbalanced 
experiments. 

The advantage of assigning subjects to several treatments is that the experiment 
can be done with fewer subjects. Thus, subject recruitment is easier, and 
financial expenses of conducting the experiment may be reduced. Another 
advantage is that the statistical analysis of the data is more sensitive because each 
subject is "matched" with himself across treatments, (p.709) 

The Model 

The study used the ARCS research-supported motivation model which considers four 
essential human characteristics and the motivational strategies associated with them (Keller, 
1987a, 1987b). They include: (1) strategies that would mobilize and sustain the 
ATTENTION of your audience, (2) strategies that would facilitate RELEVANCE to their 
personal needs or goals, (3) strategies to assist CONFIDENCE, or the audience's expectancy 
for successful understanding of contents, and (4) SATISFACTION strategies linked to the 
process or outcomes of the experience. 

Instrumentation 

A posttest analysis was used to ascertain the perceptions of employers as to whether 
or not the contents presented in NSE's existing Employer's Information Package were 
motivationally appealing or useful. Evidence of usefulness was based upon participants' 
responses to the modified Instructional Materials Motivation Survey (Keller, 1990). The 
validity of this scale's four motivational components has been supported by concurrent 
validity and field testing of the ARCS model which defines each of these components 
(Keller, 1987a, p.2). Minor wording changes to this scale were completed by this researcher 
to facilitate interpretation of information contents rather than instructional contents. As such, 
the instrument, a 36-item questionnaire, surveyed four variable employer motivations (i.e., 
attention, relevance, confidence, and satisfaction) in relation to the information contents 
given. Overall reactions to the 3 6 statements on this questionnaire were rated on a Likert 
scale (1 or A=not true, 5 or E=very true). The scoring procedure was reversed for statements 
keyed in negatively. The reliabilities of the modified scale, hereafter referred to as the 
Information Materials Motivation Survey (InfoMMS), within the March 23, 1999 reference 
study sample showed good internal consistency. The revised InfoMMS Scale proved to be 
very sensitive. While reliability of one subclass scale (Confidence) was only moderately 
high, the remaining subclass scale reliabilities were high for the Attention and Relevance 
categories, and very high for the Satisfaction category. Evidently, the InfoMMS Scale gains 
in effectiveness by the inclusion of all its content areas (subscales). Reliabilities as measured 
by Cronbach's alpha coefficient are given in Table 1 . 

Pearson's Skewness (nonsymmetry) coefficient, given by SK=3(mean - 
median)/standard deviation, for the distribution of the InfoMMS scores was found to be .41. 
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This suggests that the attribute termed motivational appeal or usefulness of contents 
presented was symmetrically distributed and approaching normality within the March 23, 
1999 baseline study sample. The fact that coefficient values must fall between -3 and 3, with 
a value of 0 yielding a perfectly symmetrical distribution undoubtedly encourages this 
finding (Freund & Williams, 1982, p.59). Accordingly, the tendency for InfoMMS raw 
scores to show a normal distribution adds considerably to any description or interpretation of 
study data. The attribute was considered a comprehensive or representative set of interrelated 
ideas pertaining to the four variable motivational categories within the InfoMMS instrument: 
Attention, Relevance, Confidence and Satisfaction. 

Statistical Analysis 

Descriptive statistics were used to determine central tendency. Missing data within a 
given dimension of the 36-item InfoMMS scale were replaced with the overall mean - i.e., 
the sample mean across all cases for a given scale item making up the scale. This was 
performed for 2 Attention subscale items (#17 and #22) in the Alternate Group for Case 
participant number 9 in the November 29, 2000 study results. Analysis of variance 
(ANOVA) was used to investigate the significance of motivational differences between 
treatment conditions that were being compared. The ANOVA replaced the t-test because "F 
and t are directly related and the analysis of variance is an extension of the test" (McCall, 
1975, p.263) to accommodate two or more groups. The index Effect Size (ES), defined as 
"the mean difference between the treated and control subjects divided by the standard 
deviation of the control group," (Smith, Glass & Miller, 1980, p.68) was used to evaluate the 
magnitude of experimental and alternate group brochure effects against the reference (focus) 
study group (used here as the "control") in standard deviation units. The reference study 
group was convened at the pre-benchmarking stage session as part of the Focus Group in 
March 23, 1999 to establish measurement baseline data of existing NSE corporate 
information materials and contents in a format (i.e., Information Materials Motivation 
Survey and Open Ended Questionnaires) for ease of subsequent comparison at the 
benchmarking stage (i.e., November 29, 2000 treatment group effects determination and 
advanced analyses). ES outcomes and standards were already reviewed by Schermer (1988) 
and adopted by this researcher to facilitate consistency in the interpretation of outcomes. In 
quantitative terms, point size estimates of less than .2 were considered "small" effects, .5 as 
"medium" in size and higher than .5 as "large." With these benchmarks in mind, an estimate 
of the effect magnitude was calculated over the posttest InfoMMS measure. In general, the 
researcher set a hypothesis presented in null form based on standard educational research 
conventions. Namely, that there would be no motivational brochure appeal difference 
between the alternate brochure and experimental brochure employers. The reader should 
know that the researcher's preferred outcome was to show no motivational brochure appeal 
difference between alternate and experimental group employers since these were newly 
designed brochures. The intent was to obtain equal appeal between both groups of readers. 
The hypothesis would be rejected or not rejected according to a standard decision rule, as 
described by Brewer (1986, p.5-23). 

If p is less than or equal to a, the Ho is rejected 

If p is greater than a, the Ho is not rejected. 
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In this context, the probability value was increased to the .10 level of significance in order to 
reduce the chances of making a Type II error. Along with manual calculations, all data were 
analyzed using the Statistical Package for the Social Sciences version 8. 



March 23, 1999 Small Focus (Reference) Group Study Results 

The InfoMMS was used to establish reference or baseline group data and to ascertain 
the overall "usefulness" of contents presented or motivational appeal engendered by the 
existing NSE Employer Information Package as perceived by the employers. Descriptive 
statistics and overall reactions to the InfoMMS questionnaire are shown in Table 1 below. 

Table 1 - Descriptive Baseline Statistics Relevant to March 23, 1999 Small Focus 
(Reference) Group Employer Session (N=7) 



Variable 


Possible 

Score 


Mean 

Score/ (%) 


SD 


Median 


Mode 


Mean 

Likert 

Rating 


Cronbach 1 s 
Alpha 


InfoMMS 


180 


126 (70%) 


22 . 16 


123 


120 


3 . 5 


0 . 9171 


Attention 


60 


38.71 (65%) 


7 .70 


38 


28 


3.2 


0 . 7673 


Relevance 


45 


29 (64%) 


7 . 72 


29 


25 


3.2 


0 . 8368 


Confidence 


45 


37.57 (84%) 


4 .47 


40 


41 


4 . 2 


0 . 6417 


Satisfaction 


30 


20.86 (70%) 


5 .55 


20 


20 


3 . 5 


0 . 9199 



Table 1 reveals that all of the components were at the moderate to high level in each 
motivational category. Together, the four motivational categories averaged 126 points 
(standard deviation = 22.16) for a moderate percentage performance of 70%. From the 
"sample percentage ± 2 Standard Errors " (Freedman, Pisani & Purves, 1978), we learn that 
the 95% confidence interval for this percentage ran from 66% to 74%. On this basis, there 
was no convincing evidence that the population percentage or percentage performance in 
other samples would exceed a high degree of acceptance or fall below a moderate level of 
acceptance should the same or similar NSE Employer Information materials continue to be 
used. Here, the "population percentage" means the percentage of all female or male 
employers between 20 to 59 years of age who prefer to read in English, have no disability 
themselves, may or may not have a family member with a disability and know someone with 
a disability. 

The author labeled percentages between 0 and 50 as very low, 51 to 70 as low, 71 to 86 as 
high and above 86 as very high for the purposes of educational research. These short 
classifying phrases facilitated consistency in the interpretation of acceptance of brochure 
contents (i.e., direction and intensity of appeal) or percentage performances for the 
instrument used in this study. 
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BENCHMARKING STAGE 

November 29, 2000 Small Alternate Group and Experimental Group Study Results 

The InfoMMS was used to ascertain the overall "usefulness" of contents presented or 
motivational appeal engendered by the Alternate and Experimental Group Employer 
brochures. The InfoMMS scale surveyed four variable employer motivations (i.e., attention, 
relevance, confidence, and satisfaction) in relation to the contents given. Overall reactions to 
the four motivational categories contained within the InfoMMS questionnaire were rated on 
a Likert scale (1 or A=not true; 5 or E=very true). The resulting performance levels (i.e., 
uppercase or capitalized for Alternate Group and lowercase for Experimental Group) were 
compared in terms of the scale's four motivational components and portrayed on an inverted- 
U curve as Figure 1 below. 

COMBINED AUDIENCE CHARACTERISTICS 




Motivational Appeal (Usefulness) Level 
(Mean ratings on 5-point Likert scale) 

Figure 1. November 29, 2000 InfoMMS POSTTEST GRAPH SHOWING MOTIVATIONAL APPEAL 
OF ALTERNATE GROUP (N=7) AND EXPERIMENTAL GROUP (N=7) BROCHURE ACCEPTANCE LEVELS AS 
PERCEIVED BY EMPLOYERS 

It reveals that the Alternate Group Employer brochure mobilised and sustained the 
Attention of employers to a high degree (of acceptance). The 49.9 average (or 4.2 Likert 
rating) along this dimension translated into a high percentage performance of 83%. The 
contents were Relevant to their personal needs or goals to a high degree to an established 
average of 34.1 points (3.8 Likert rating) or a percentage performance of 76%. Confidence, 
or the employers' expectancy for successful recall or understanding of contents averaged a 
very high 40.6 (4.5 Likert rating or 90%), and Satisfaction with the process or outcomes of 
the experience averaged a high 22 (3.7 Likert rating or 73%). It is of interest that all of the 
components were at high to very high levels in each motivational category. None of the 
employers were undermotivated in the process or as a result. Together, the four motivational 
categories averaged 146.57 points (standard deviation = 13.58) for a high percentage 
performance of 8 1 %. 
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The Experimental Group Employer brochure mobilised and sustained the Attention 
of employers to a high degree (of acceptance). The 43.6 average (or 3.6 Likert rating) along 
this dimension translated into a percentage performance of 73%. The contents were Relevant 
to their personal needs or goals to a high degree to an established average of 33 points (3.7 
Likert rating) or a percentage performance of 73% as well. Confidence, or the employers' 
expectancy for successful recall or understanding of contents averaged a high 36.6 (4.1 
Likert rating or 81%), and Satisfaction with the process or outcomes of the experience 
resulted in a low performance averaging 18.4 points (3.1 Likert rating or 61%). Motivational 
performance levels ranged from low to high. Together, the four motivational categories 
averaged 131.57 points (standard deviation = 16.98) for a high percentage performance of 
73%. 

The overall Alternate Group InfoMMS sample mean 146.57 or overall sample 
percentage 81% estimates the average of a population, but to indicate the precision of this 
estimate this researcher gave a confidence interval. From Freedman, Pisani, & Purves (1978, 
p.345) we learn that the interval "sample percentage ± 2 Standard Errors" has a confidence 
level of about 95%. The 95% confidence interval for this percentage ran from 75% to 88%, 
as illustrated in Figure 2 below. Thus, the researcher can be reasonably confident of 
detecting a high to very high overall percentage performance between 75% and 88%. 



2SE 2SE 





75% 81% 88% 

Sample 
percent 



Figure 2. 95% Confidence interval on overall Alternate Group InfoMMS 

Sample Percentage. 



The range of this estimate is exact given a normal population distribution and "... is 
approximately correct for large n in other cases" (Moore & McCabe, 1989, p.512). On this 
basis, there was no convincing evidence that the population percentage or percentage 
performance in other samples would exceed a very high degree of acceptance or fall below a 
high level of acceptance should the same or similar Alternate Group Employer brochure 
materials continue to be used. Here, the "population percentage" means the percentage of all 
female or male employers between 20 to 59 years of age who prefer to read in English, have 
no disability themselves, may or may not have a family member with a disability and know 
someone with a disability. 

The overall Experimental Group InfoMMS sample mean 1 31.57 or overall sample 
percentage 73% estimates the average of a population, but to indicate the precision of this 
estimate this researcher gave a confidence interval here as well. The 95% confidence interval 
for this percentage ran from 64% to 82%, as illustrated in Figure 3 below. Thus, the 
researcher can be reasonably confident of detecting a low to high overall percentage 
performance ranging between 64% and 82%. 
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^ ^ ^ 



64% 73% 82% 

Sample 

percent 

Figure 3. 95% Confidence interval on overall Experimental Group InfoMMS 

Sample Percentage. 

Accordingly, there was no convincing evidence that the population percentage or percentage 
performance in other samples would exceed a high degree of acceptance or fall below a low 
level of acceptance should the same or similar Experimental Group Employer brochure 
materials continue to be used. The low performance in Satisfaction appears to undermine the 
overall high percentage performance of the Experimental Group Employer brochure. Here, 
the "population percentage" means the percentage of all female or male employers between 
20 to 59 years of age who prefer to read in English, have no disability themselves, may or 
may not have a family member with a disability and know someone with a disability. 

SUPPLEMENTARY RESULTS 

Observed Significance Levels 

Table 2 

ANOVAs for the Posttest on Information Materials Motivation Survey (InfoMMS) as a 

Dependent Variable 





Sum of 
Squares 


Mean 

Square 


df 


F 

Value 


P 


InfoMMS Between Groups 


787.500 


787.500 


1 


3.333 


0.093 * 


Within Groups 


2835.429 


236.286 


12 








3622.929 




13 






A Between Groups 


138.286 


138.286 


1 


4.478 


0.056 * 


Within Groups 


370.571 


30.881 


12 












13 






R Between Groups 


4.571 


4.571 


1 


.282 


0.605 


Within Groups 


194.857 


16.238 


12 








199.429 




13 






C Between Groups 


56.000 


56.000 


1 


6.497 


0.026 * 


Within Groups 


103.429 


8.619 


12 








159.429 




13 






S Between Groups 


44.643 


44.643 


1 


1.957 


0.187 


Within Groups 


273.714 


22.810 


12 








318.357 




13 








* Significant at the .10 alpha level 
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Only 14 observations were used in this analysis. The omnibus analysis of variance 
revealed a significant difference by group, df=l, F=3.333, p=0.093. It can be seen in Table 2 
on the analysis of variance that the group variable was significant at the .10 alpha level. 
Thus, the null hypothesis of no motivational brochure appeal difference between the 
alternate and experimental conditions was rejected according to the decision rule. 
Supplementary ANOVA results also showed significant differences between the groups on 
the Attention (p=0.056) and Confidence (p=0.026) sub-component categories. The analysis 
did not reveal any significant differences for the Relevance (p=0.605) and Satisfaction 
(p=0.187) sub-component categories. 

Effect Sizes 

The initial observations show that the two treatment brochures differ significantly in 
usefulness but tell us little about their effectiveness. On this basis, it is premature to conclude 
the treatments (that is, the newly designed experimental and alternate brochures and 
contents) had any effect. So, to estimate how large the treatment effects could really be, this 
researcher compared the average effect sizes for the Alternate and Experimental Groups 
against the reference data from the Focus Group session which was used as the baseline 
"control." These are outlined in Table 3 below. 

Table 3 

SPSS Summary Statistics for Post Study Groups on Information Materials Motivation Survey 
(InfoMMS) as a Dependent Variable versus Reference (Focus) Study "Control" Group 





Reference 
"Control" Group 

(N=7) 

March 23 1999 Study 


Alternate Group 

(N=7) 

Nov 29, 2000 Study 


Experimental Group 

(N=7) 

Nov 29, 2000 Study 


Attention Factor 


X=38.7 


X=49.86 


X=43.57 




SD=7.70 


SD=4.74 


SD=6.27 






ES=1.5 


ES=.63 


Relevance Factor 


X=28.8 


X=34.14 


X=33 




SD=7.72 


SD=3.58 


SD=4.44 






ES=.69 


ES=.54 


Confidence Factor 


X=37.6 


X=40.57 


X=36.57 




SD=4.47 


SD=3.51 


SD=2.23 






ES=.66 


ES=.23 


Satisfaction Factor 


X=20.9 


X=22 


X=18.43 




SD=5.55 


SD=4.44 


SD=5.09 






ES=.2 


ES=-.45 


IMMS Totals 


X=126 


X=146.57 


X=131.57 




SD=22.16 


SD=13.58 


SD=16.98 






ES=.93 


ES=.25 
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Effects of Alternate Group against Reference "Control" Group 

There was an overall effect size gain of .93 standard deviation units - a large effect. 
Thus, the ES of .93 also demonstrates that Alternate Group InfoMMS change has been 
increased to a large degree. An estimate of the effect magnitude was calculated over the 
posttest InfoMMS measure, per motivational usefulness factor as well. Effect size gains were 
distributed as follows: 1.5 standard deviations (SDs) for Attention - a large effect, .69 for 
Relevance - a large effect, .66 for Confidence - a large effect, and .2 SDs for Satisfaction - 
approaching a small effect. More importantly, these measures consistently reflect some gains 
in each of the brochure content areas. 

The immediate overall effect size gain of .93 on the InfoMMS scale also means that 
the "average" employers receiving the untreated baseline brochure who scored at the 50 th 
percentile level would see their usefulness (or motivational appeal) scores rise to the 82 nd 
percentile level if provided the alternate treatment brochure. This represents a gain of 32 
percentile points over the untreated "control" condition. It also suggests that the overall 
effect of the alternate treatment brochure is very engaging. As for interpreting effect sizes by 
way of percent of nonoverlap (See Cohen, 1988, pp.21-23) of the alternate treated group's 
scores with those of the untreated baseline "control" group, the ES of .93 indicated a 
nonoverlap of about 52% in the two distributions. 

Effects of Experimental Group against Reference "Control" Group 

There was an overall effect size gain of. 25 standard deviation units - a moderate 
effect. Thus, the ES of .25 also demonstrates that Experimental Group InfoMMS change has 
been increased to a moderate degree. As well, an estimate of the effect magnitude was 
calculated over the posttest InfoMMS measure, per motivational usefulness factor. Effect 
size gains were distributed as follows: .63 standard deviations (SDs) for Attention - a large 
effect, .54 for Relevance - a large effect, .23 for Confidence - a moderate effect, and -.45 
SDs for Satisfaction - a negative effect. Except for the moderately negative Satisfaction 
measure, these data consistently reflect some gains in each of the brochure content areas. 

The immediate overall effect size gain of .25 on the InfoMMS scale implies a 50 th to 
the 60 th percentile gain in the direction and intensity of usefulness (or motivational appeal) 
scores by the average experimental treatment brochure employers. This represents a gain of 
only 10 percentile points over the untreated "control" condition. It also suggests that the 
overall effect of the experimental treatment brochure is not that motivating. Interpreting the 
effect size by way of percent of nonoverlap of the experimental treated group's scores with 
those of the untreated baseline "control" group, the ES of .25 indicated a nonoverlap of about 
18% in the two distributions. 



DISCUSSION OF RESULTS AND FINDINGS 
Summary of Overall Study Findings 

There appear to be four major findings in this exploratory pilot study. The overall 
findings of this study were that 
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1 . In general, both newly designed treatment brochures experienced positive effect 
size gains when compared to the untreated baseline "control" condition contents. 

2. The immediate overall effect size gain of .25 on the experimental treatment 
brochure InfoMMS scale implies a 50 th to the 60 th percentile gain in the direction 
and intensity of usefulness (or motivational appeal) scores by the average 
experimental treatment brochure employers. The gain of only 10 percentile points 
over the untreated "control" condition suggests that the overall effect of the 
experimental treatment brochure was not that motivating. The ES of .25 also 
indicated a nonoverlap of about 18% of the experimental treated group's scores 
with those of the untreated baseline "control" group. 

3. There was an immediate overall effect size gain of .93 on the alternate treatment 
brochure InfoMMS scale. This means that the "average" employer receiving the 
untreated baseline brochure who scored at the 50 th percentile level would see their 
usefulness (or motivational appeal) scores rise to the 82 nd percentile level if 
provided the alternate treatment brochure. The gain of 32 percentile points over 
the untreated "control" condition suggests that the overall effect of the alternate 
treatment brochure was very engaging. The ES of .93 also indicated a nonoverlap 
of about 52% of the alternate treated group's scores with those of the untreated 
baseline "control" group. 

4. Supplementary ANOVA results showed a significant difference by group. This 
confirms that employers who received the alternate treatment brochure perceived 
it as significantly more useful (or motivationally appealing) than the experimental 
treatment brochure. 

Observations 

The overall F-test result showed significant differences between the Alternate Group 
employers' and Experimental Group employers'. From the point of view of the ARCS model, 
employers perceived Alternate Group materials and contents much more positively because 
they significantly mobilized and sustained their attention by a much higher degree, the 
contents were perceived as more relevant to their personal needs or goals, their confidence 
levels for successful understanding of contents were significantly higher and as a result, they 
were m ore s atisfied w ith t he p rocess o r o utcomes o f t his e xperience. Thus, there is some 
support for Arredondo's (1991, p.22) truism that "if it doesn't get and keep their attention, it 
can't be meaningful, memorable, or activating." The proportionally greater effect size gains 
in all ARCS model subcomponent motive factors favouring the Alternate group support this 
view. Effect size gains for the Alternate Group brochure and its contents as a whole was .93 
standard deviation units (- a large positive impact with about 52% nonoverlap) compared to 
.25 standard deviation units (-a slightly moderate positive effect with about 18% nonoverlap) 
for the Experimental Group brochure. In retrospect, while both brochures engendered greater 
"usefulness" or motivational appeal among employers than the initial Baseline "Control" 
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Group employer information materials, it seems clear that the results favoured the alternate 
brochure group of employers. 

Conclusions and Recommendations 

The results of this exploratory study may have practical implications. It has been 
found that corporate produced information materials can be more readily accepted into a 
client's motivational schema if the target audience itself has input into the change (or 
learning) process. As such, it is not inconceivable that a wider degree of generality may 
eventually be found in similar research situations and samples. Given the complexities of 
corporate, employer and researcher values and the prohibitive costs of undertaking 
educational research, it is recommended that a focus group be used as a qualitative 
supplement to any corporate decision-making change effort dealing with formative 
evaluation. The focus group gave useful feedback and suggestions for revision of existing 
materials and contents. This phase of the study provided vital internal pre-benchmarking 
stage "pretest" data for (1) estimating the reliability of the InfoMMS questionnaire, (2) 
estimating the performance levels or usefulness of existing corporate materials, (3) clarifying 
design, development and use of future corporate information materials and preferences, and 
(4) providing a ready bank of baseline "control" data from which future materials and 
associated benchmarking partner data can be compared against. Researchers, practitioners 
and program planners are encouraged to use these findings in conjunction with a focus group 
in the design of future activities related to corporate information materials and contents 
development. For all practical intends, formative production and evaluation efforts should 
seriously consider using multiple samples and repeated testing procedures to ensure 
materials are interpreted in the way intended. Central to this post-benchmarking process is 
the calculation and reporting of effect sizes which provide objective and standardized 
measures of mean differences attributed to the exposure of groups treated to a modification 
in comparison to a reference (baseline) or "control" group that does not receive a quality 
design change opportunity. Newly established effect sizes should be discussed for 
effectiveness and further usefulness. The results of these analyses showed that the 
experimental brochure and the alternate brochure in particular were more useful than the 
baseline "control" brochure. The degree of usefulness was represented by an effect size of 
.95 for the Alternate Group and .29 for the Experimental Group, both of which were positive. 
For the larger effect size, the difference in the distributions was obvious and there was very 
little overlap. For the smaller effect size, the two distributions overlap a great deal and the 
effect was difficult to see. Accordingly, the experimental brochure had a weaker effect and is 
susceptible to further instability. Thus and so, the alternate group effect sizes are 
recommended as the re-calibrated standard (i.e., updated future reference benchmark). 
Alternate group data showed that the employer was more likely to react positively on the 
posttest InfoMMS measure if the employer performs in the high range on each 
subcomponent ARCS factor. Additional supporting evidence showed that the 95% 
population percentage for the Alternate Group was 81%, and the likely range was 75% to 
88%. In other words, the percentage performance in other samples would not tend to fall 
below a high level of acceptance should the same or similar Alternate Group employer 
brochure materials continue to be used. This tendency can be verified by a larger sample of 
subjects to provide a narrower precision of the likely range. However, unlike significance 
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tests, effect-size statistics are not dependent on sample size which makes their reporting 
essential to any research. It has been shown that to remain successful, promotional materials 
and contents need to achieve and maintain motivational acceptance standards in the high 
range, but only continued review processes will ensure continued effectiveness and 
usefulness. 

In the final analysis, the presence of a negative aftereffect on audience response 
might be real. Such effects may or often go unnoticed on the radar screen because they tend 
to occur at the subcomponent level as opposed to a global one. So, it is incumbent upon 
designers and researchers to measure for these aftereffects or to measure the phenomenon 
susceptible of curtailing ill effects in favour of improving product and materials effectiveness 
or usefulness. Those who are dazzled by "attractive" or "smart" looking materials and 
contents may unintentionally markdown the "tame" looking ones when the latter may 
produce the most desirable effect/s. More research is needed to explain the sometimes 
uneasy choices that an audience makes. This study speaks to that void. The use of 
descriptive and inferential statistics, corporate advisory committees and focus groups each 
have something equally valuable to contribute to product development, and therefore should 
be used in conjunction with each other. The general conclusion to be drawn from this 
exploratory formative evaluation study is that the same or similar uses of internal 
benchmarking and statistical analysis provide one avenue to fostering quality in the 
motivational design of corporate information materials. 
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And Brochure Descriptions 




17 



17 



Table 4. 

Descriptions of Open Ended Survey and Qualitative Portion of March 23, 1999 Small 
Focus (Reference) Group Employer Session (N=7) Data by Frequency of Occurrence 



What did you like best about the existing NSE Employer Information 
Package? (8 1/2" x 11" format) 

Accomplishments/Achievements (N=2 ) 

Comprehensive Information (N=2) 

Honesty/Real Life Examples (N=2) 

History of Org. (N=2) 

Mission Statement (provides objectives to work with) (N=2) 

Employer's responsibili ties/Needs of Participants (N=2) 
Systematic/Logical Sequence/Methodology (N=2) 

Easy reading (N=2) 

Newsletter (N=l) 

Frequently Asked Questions by Employers (N=l) 

Liked it all (N=l) 

What did you like least about it? 

Repetition (N=2) 

Literary usage lacks passionate appeal (N=2) 

About NSE info, introduced to late (N=l) 

First two pages somewhat daunting, tempted to skip them (N=l) 

The organisation chart, role, reporting, etc., unclear (N=l) 

Few graphics (N=l) 

Few examples (N=l) 

Design (N=l) 

Better understanding of message/ inf ormation required (N=l) 

No uniqueness in service (N=l) 

Did not like much at all about it. (N=l) 

Employer Recommendations for Improvement 

Include Pictures/Graphs/Illustrations and/or the like (N=3) 
List/Describe Successes/Testimonials (N=2) 

List Current Employers/Companies using programs (provides prospective 
employers greater feeling of confidence) (N=2) 

About NSE info up front (N=l) 

Impact of NSE (N=l) 

Use colour (N=l) 

Back to back use of Information on Sheets (N=l) 

Supply Employer references at time of application (N=l) 

Use greater appeal in contents (N=l) 

Uniqueness/differences in services (N=l) 

Format Preference 

Pamphlet/Brochure = (N=3) 

Existing 8 1/2 x 11" = (N=3) 

No Response (N=l) 



o 

ERIC 



.18 



BEST COPY AVAILABLE 



18 



Facilitator Face to Face Portion of Focus Group Session Results 

Package is easy to read 

Package is well organised, it evolves or follows a logical order 
Content should be condensed 

More examples of clients we work with, less info on each company, but 
more company examples 
Get endorsements from clients 

Companies interested in knowing companies have used our services 
Mention how long we have worked with a company, how long consumer 
employed by company 

Examples/testimonials spread throughout document 
Use company logos 

Remove repetition from literature 
Use more visuals 

Info package was clear, it tells you what you need to know 
Purpose of the org has to 1 st thing you see 
Use references 

Network South Logo/mission statement should be on front of the folder 

Box important information 

Include info on screening applicants 

Do they fill out application forms (give example) 

Do we have evaluations (give example) 
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Some Elements Incorporated Into The Experimental Brochure 
Panel 1 Outside Front Cover (Far Right) 

• 3/4 " corporate logo (left justified) , name of corporation (right 
justified) alongside corporate logo 

• colours and hues of corporation adopted throughout 

• single message tag line (right justified 3 inches below logo. It 
reads "Connecting your business to supported employment services" 

• Also 3 black and white 1" x 1 1/4" pictures positioned below tag 
line vertically aligned and separated by 1/8" space each 

• Bottom of brochure caption reads "An Information Guide For 
Employers" 

Panel 2 Left Inside Cover 

• Who Are We? 1/4" Header followed by information 

• Philosophy 1/8" Sub-header and information 

• Mission 1/8" Sub-header and information 

• History 1/8" Sub-header and information 

Panel 3 Middle of Inside Cover 

• Why Hire From Us? 1/4" Header 

• Benefits 1/8" Sub-header and information 

• Employer Services 1/8" Sub-header and bulleted information 

• Staff 1/8" and information 

Panel 4 Far Right Inside Cover 

• Two employer endorsements with statements enclosed in shaded box 

• Programs 1/4" Header with bulleted information 

Panel 5 Outside Cover (Far Left) 

• Our Employer Partners Have Included... (Employers listed) 

Panel 6 Middle Outside Cover 

• Design and Layout donated by acknowledgements (top of panel) 

• Printing Courtesy of acknowledgements (top of panel) 

• 1/2" corporate logo bottom of panel with corporate name and contact 
address points etc... 

Note: Inside Cover contents positioned NSE Inc. as experts in the 
field. 
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Some Elements Incorporated Into The Alternate Brochure 

Panel 1 Outside Front Cover (Far Right) 

• line bar on top on top of panel 

• 2" x 2" corporate logo (top center location) 

• colours and hues of corporation adopted throughout 

• single message tag line (5 1/2" below top of panel) . It reads 
"Building Positive Relationships With Local Business" 

• Bottom of panel caption spells out corporation's name 

• Line bar on bottom of panel 

Panel 2 Left Inside Cover 

• Mission Statement 1/4" Header only 

• Philosophy 1/8" Sub-header and information 

• Role and Mission 1/8" Sub-header and information 

• Here Are The Facts: 1/8" uppercase Sub-header followed by 3 bullets 
of information 

Panel 3 Middle of Inside Cover 

• Network South Enterprises 1/2" Header across top of panel 

• Here's How We Work uppercase 1/8" Sub-header followed by 4 bullets 
of information 

• Soft Watermark of corporate logo spanning panels 2 through 4 

• Employer Services 1/8" uppercase Sub-header followed by 2 bullets of 
information 

Panel 4 Far Right Inside Cover 

• Employer Benefits 1/8" uppercase sub-header and small description 
followed by 4 bullets of information 

• Contact information bottom of panel 

Panel 5 Outside Cover (Far Left) 

• Here is what people have to say about Network South Enterprises Inc. 
Statement on top of panel followed by 1 Employer endorsement, 1 
Parent endorsement, 1 NSE Graduate endorsement and 1 NSE Employment 
Consultant endorsement 

Panel 6 Middle Outside Cover 

• Brochure Set-Up and Design Courtesy of Acknowledgements (top center 
of panel) 

• Brochure Printing Courtesy of acknowledgements (center of panel) 

• NSE corporate name spelled out in addition to corporate support and 
sponsorship acknowledgement statements (bottom of panel) 

Note: Inside Cover contents used less descriptive information 
statements and relied more on bulleted information points . No pictures 
or visuals outside the watermark supported the contents . 
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